For the current water conservancy dams mainly rely on manual on-site inspections， which have high operating costs and low efficiency， an improved detection algorithm based on YOLOv5 was proposed. Firstly， a modified multi-scale visual Transformer structure was used to improve the backbone， and the multi-scale global information associated with the multi-scale Transformer structure and the local information extracted by Convolutional Neural Network （CNN） were used to construct the aggregated features， thereby making full use of the multi-scale semantic information and location information to improve the feature extraction capability of the network. Then， coordinate attention mechanism was added in front of each feature detection layer of the network to encode features in the height and width directions of the image， and long-distance associations of pixels on the feature map were constructed by the encoded features to enhance the target localization ability of the network in complex environments. The sampling algorithm of the positive and negative training samples of the network was improved to help the candidate positive samples to respond to the prior frames of similar shape to themselves by constructing the average fit and difference between the prior frames and the ground-truth frames， so as to make the network converge faster and better， thus improving the overall performance of the network and the network generalization. Finally， the network structure was lightened for application requirements and was optimized by pruning the network structure and structural re-parameterization. Experimental results show that on the current adopted dam disease data， compared with the original YOLOv5s algorithm， the improved network has the mAP （mean Average Precision）@0.5 improved by 10.5 percentage points， the mAP@0.5：0.95 improved by 17.3 percentage points； compared to the network before lightening， the lightweight network has the number of parameters and the FLOPs（FLoating point Operations Per second） reduced by 24% and 13% respectively， and the detection speed improved by 42%， verifying that the network meets the requirements for precision and speed of disease detection in current application scenarios.